Recent years have seen drastic increases in the use of portable computing devices, such as smart phones and tablet computers. Today's consumers are utilizing such devices for a wide variety of different purposes, such as to access and search the Internet, purchase products and services, capture and/or send digital images, compose electronic mail (email) messages, make telephone calls and the like. Many of such computing devices include one or more cameras that are capable of capturing images of the user's face, among other things. Some devices even include software that attempts to detect, track and/or recognize the user's face for many different reasons. For example, it may be useful for the device to determine the position of the user's face to determine whether the user is looking at the display screen so that the device may choose to perform various actions based on that knowledge. In order to perform this, the device needs to first detect whether an image (i.e., portion thereof) contains the user's face. However, facial detection is not a trivial task and many conventional facial detection techniques are imperfect due to the wide range of objects that may be contained in an image and due to the limited time frame that is usually available for detecting the presence of a face.